Preliminary landscape analysis of deep tomographic imaging patents

Over recent years, the importance of the patent literature has become increasingly more recognized in the academic setting. In the context of artificial intelligence, deep learning, and data sciences, patents are relevant to not only industry but also academe and other communities. In this article, we focus on deep tomographic imaging and perform a preliminary landscape analysis of the related patent literature. Our search tool is PatSeer. Our patent bibliometric data is summarized in various figures and tables. In particular, we qualitatively analyze key deep tomographic patent literature.


Introduction
With the advances of artificial intelligence (AI), deep learning (DL) has emerged as a mainstream approach with successful applications in many areas. Since 2016, deep reconstruction or deep imaging methods have been actively developed, especially in the field of medical imaging [1][2][3]. Promising results on medical imaging are widely reported on diverse topics ranging from data acquisition and processing [4], image reconstruction and enhancement [5,6] to radiomics and health analytics [7], and more. Clearly, AI/DL is paving an exciting way to improve or innovate medical imaging devices, and diagnostic and therapeutic procedures. With rigorous and systematic assessment and regulation, AI imaging software and devices may assist or compete effectively with radiologists, eventually transforming the current model of medical and healthcare practice in various aspects.
According to a 2020 press release from the Yole Group (https:// www. yoleg roup. com/ press-relea se/ medic al-imagi ng-artif icial-intel ligen ce-chang es-the-rules/), "AI has the potential to change all of our diagnostics and treatment procedures to enable more personalized and effective medicine. " "At Yole, we estimate the total market in 2025 for software generated revenues through the sale of AI tools will reach US$2.9 billion with a 36% CAGR (Compound Annual Growth Rate) between 2019 and 2025 (Fig. 1). These revenues can be shared between the main applications including improved image capture, noise reduction, image reconstruction, screening, diagnostic and treatment planning. " Given the huge commercial potential of AI-based imaging technologies, intellectual property plays an increasingly important role in the imaging industry, research and user communities. In this context, for researchers and developers, patent landscape analysis and literature review are indispensable. To understand the landscape of AI-based deep tomographic imaging technologies that promise to be clinically relevant, we are motivated to survey relevant patent literature over the past decade or so. As used herein, "patent literature" includes issued patents and published patent applications (i.e., pre-grant patent application publications). As further used herein, "patent documents" ("PGPubs") are also used to mean patent literature. The rest of this paper is organized as follows. In the second section, we present our methodology. Our main patent search tool is PatSeer [8]. Our search strategy and analysis methods are described in this section. In the third section, we present our PatSeer search results in figures and tables. While the third section is largely datadriven, in the fourth section we analyze key deep tomographic patent documents aided by citation analysis. In the last section, we discuss relevant issues and conclude the paper.

Methods
Over recent years, the importance of patent literature has become more recognized in the academic setting, as evidenced by the establishment of the National Academy of Inventors in 2010. In the context of AI, DL, and data science, patents are relevant to not only industry but also academe and other communities. To perform our landscape analysis of the patent literature on deep tomographic imaging, we used the patent search and analysis tool PatSeer. We note that other patent search and analysis tools are also available including, but not limited to, AcclaimIP (available from Anaqua), LexisNexis TotalPatent One ® , etc.
Specifically, the tool we used is PatSeer ProX, developed by Gridlogics. PatSeer ProX includes big-data analytic methods and performs relatively fast through, in our opinion, a very user-friendly interface. We find that the search rules used by PatSeer ProX are like those used by Scopus. According to the PatSeer website, the Pat-Seer ProX patent database covers more than 136-million patent publications, and 96-million full-text records. Furthermore, PatSeer maintains a scalable big-data platform with AI-based semi-automated algorithms to process and analyze raw data from over 300 sources.
As an illustration, with the "deep learning" as the search phrase in the title, abstract and claim fields from

Visualization of Patents on Deep Tomographic Imaging
In this section, we focus on our patent search results on AI-based medical imaging techniques. Initially, we performed keyword-based searches in the title, abstract and claim fields but we obtained quite many irrelevant results. For example, the inclusion of claims included French PGPub FR2733596B1 [9] which only mentions 'Ultrasound' in the claims. As another example, International PGPub WO0124700A1 [10] is for fingerprint detection. As a result, our searches were limited to the title and abstract fields only. After further deliberation, we defined the following search expression: TA:(("machine learning" or "deep learning" or "deep nets" or "neural network" or "deep network" or "deep neural network" or "artificial intelligence") and ((raw data process* or "k-space data" or "tomographic data" or "sinogram" or image reconstruct* or post-process*) or ("image quality" or "artifact reduction" or "low dose scan" or "fast scan" or "under sample" or "noise reduction")) and (medical imag* or tomograph* or "CT" or "computed tomography" or "PET" or "positron emission tomography" or "SPECT" or "single photon emission computed tomography" or "nuclear imaging" or "MRI" or "magnetic resonance" or "Ultrasonography" or "ultrasound" or "optical coherence tomography" or "OCT") and not ("display apparatus" or assess*)), where TA means in the title and abstract fields. Note that the filter function, not ("display apparatus" or assess*), was empirically added to exclude irrelevant hits; for example, DE69031523D1 [11] matches the other parts of the overall expression but only describes a method to optimize a display window, and US8086007B2 [12] describes an image quality assessment method. In total, this search yielded 757 records. Figure 4 summarizes the numbers of patents per application year grouped by the original assignee.    US20180018757A1 describes a method to improve the quality of the projection data using machine learning (ML) models. International PGPub WO2017223560A1 [13] describes ML-based tomographic/tomosynthetic techniques that use a neural network that includes more than three layers and can be applied to either raw data or initial image domains. PGPub US2020034998A1 [14] is an example of DL for MRI imaging, where a neural network model is used with data consistency.
To assess the quality of an issued patent or published patent application quantitatively, PatSeer provides a 360° Quality metric, which is a weighted average of 4 contributing scores. The contributing scores include Citation Quality (CQ), Market Quality (MQ), Legal Quality (LQ) and Document Quality (DQ) [8]. CQ considers the number, recency and type of forward citations of a patent document. MQ is based on the patent family's global market coverage. LQ reflects how aggressive the company protects the patent family. DQ measures the intrinsic quality of a patent based on its structural elements. Then, the portfolio value index is defined as the sum of the 360 Quality scores of all the patent families present in a portfolio, which, according to Gridlogics, is roughly proportional to the overall realizable value of the portfolio. Figure 6 shows the most valuable portfolios, according to the PatSeer Quality metric. Among them, Generic Electric Co (GE), Canon Inc and Siemens AG hold the most valuable portfolios as industry leaders, while Rensselaer Polytechnic Institute (RPI), Zhengzhou University and Stanford University are the academic leaders.

Analysis of Patents on Deep Tomographic Imaging
We work in the medical imaging field with diverse interests but with an emphasis on computed tomography (CT). Given the broad view presented in the preceding section, in this section we analyze further deep CT reconstruction patent literature and other related technologies. This analysis provides a unique perspective that will help us plan future research and development activities. In the following sub-sections, we analyze patent literature based on specific applications and select representative patent documents for analysis. Tables 1, 2 and 3 list related patent documents and further include brief comments.

Tomographic image reconstruction
DL has significant implications for tomographic image reconstruction, as first described in our perspective paper on deep imaging [1]. Our perspective paper presents three specific examples with simulation results; i.e., CT image super-resolution, singogram quality improvement, and CT image denoising. More generally, when a dataset is truncated, for example, in cases of limitedangle, few-view and interior or local reconstruction, or an image is distorted or comprised by artifacts, including metal artifact, scatter artifact, motion artifact and beamhardening effect, a synergistic combination of conventional tomographic methods and DL-based imaging may enhance image quality and diagnostic performance. An international patent application (PGPub WO2017223560A1) [13] was filed with a priority date of June 24, 2016, describing tomographic image reconstruction systems and methods based on DL techniques. This PGPub discloses a general framework for image reconstruction from raw data directly or from an intermediate reconstructed image with the DL/ ML approach, suggesting a potential for deep tomographic reconstruction to surpass classic reconstruction algorithms.
Another published international patent application (PGPub WO2019074879A1) [15] is directed to image reconstruction with DL/ML. The approach involves a ML model training process and use of the trained model for image generation. The ML model training may use multiple images generated from a single set of tomographic projections or images. The trained model may be used to generate a final image from the projection data, which is a less computationally intensive algorithm. The issued patent US10475214B2 [16] uses ML to solve tomographic reconstruction problems, which are usually in large scale and are space variant while correcting various artifacts. International PGPub WO2018126396A1 [17] presents DL-based raw data correction and estimation for tomographic reconstruction.
International PGPub WO2019060843A1 [18] discloses an image reconstruction method using a ML regularizer. Specifically, an iterative reconstruction technique can incorporate a ML model as a regularization filter for the image reconstruction. International PGPub WO2018236748A1 [19] describes an image reconstruction scheme using DL/ML for tomographic imaging. The method first produces a plurality of intermediate images using iterative reconstruction method and furthermore  US2021012541A1 Apparatus and method using DL to improve analytical tomographic image reconstruction A method is proposed to improve the image quality of images generated by analytical reconstruction of a CT image. This improved image quality results from a DL network that is used to filter a sinogram before back projection but after the sinogram has been filtered using a ramp filter or other reconstruction kernel.
Canon 07-11-19 US2021192809A1 Tomographic image reconstruction using AI engines A method includes obtaining two-dimensional (2D) projection data and processing the 2D projection data using the AI engine. AI engine may involve: generating 2D feature data by processing the 2D projection data using the multiple first processing layers, reconstructing first three-dimensional (3D) feature volume data from the 2D feature data using the back-projection module; and generating second 3D feature volume data by processing the first 3D feature volume data using the multiple second processing layers. US2021118204A1 Method for reconstructing incomplete data of X-ray absorption contrast computed tomography based on DL A DL-based method is proposed using a filtered back projection (FBP) algorithm to obtain an initial reconstructed image; forward projecting the initial reconstructed image to obtain artifact-contaminated complete projection sequences; using a DL technique to process the artifact-contaminated projection sequences to obtain artifact-free projection sequences; using the FBP algorithm to reconstruct the artifact-free projection sequences to obtain a final reconstructed image.

WO2019063760A1 DL based scatter correction
A neural network is trained on Monte Carlo simulated imaging data with at least one X-ray photon scattering mechanism to convert the projection data to a scatter free data, which is further used to reconstruct the CT image.
Philips 09-28-17  US2021192810A1 Tomographic image analysis using AI engines A method is proposed by obtaining first 3D feature volume data and processing the first 3D feature volume data using an AI engine that includes multiple first processing layers, an interposing forward-projection module and multiple second processing layers.

12-20-19
WO2018232388A1 Systems and methods for integrating tomographic image reconstruction and radiomics using neural networks A ML framework is used to either integrate tomographic images and radiomics or map tomographic raw data directly to diagnostic information.
RPI 06-16-17 US2020311878A1 Apparatus and method for image reconstruction using featureaware DL A method is proposed to use a neural network to perform on a feature-aware reconstruction. A neural network model is trained to perform feature-aware reconstruction using a target image is spatially-dependent degree denoised and artifact suppressed based on the features in the images.
Canon 04-01-19 WO2020214911A1 Method and system for generating attenuation map from single photon emission computed tomography (SPECT) emission data based upon DL A system is proposed to use ML models to estimate attenuation coefficients from only SPECT emission data.  [20] discloses a technique for improving projection data via DL/ML. The method transforms projection data from low-quality projection to higher quality, and performs image reconstruction from the improved projection images. A DL model is trained from matched pairs of lower-quality (lower-dose) and corresponding higher-quality (higherdose) projection data and used to transform lowerquality (lower-dose) projection data to higher-quality (higher-dose) ones after training is finished. PGPub US2020311878A1 [21] describes a feature-aware DL method for image reconstruction. A neural network model is trained to perform feature-aware reconstruction using a target image for spatially-dependent denoising and artifact suppression.
The last technology we comment on is virtual monochromatic CT image reconstruction. International PGPub WO2019067524A1 [22] discloses processing current-integrating data and images via ML to produce virtual monochromatic images. The neural network is configured to learn a nonlinear function to map a CT image reconstructed from projection data collected in a current-integrating X-ray detector to an image reconstructed from a virtual monochromatic projection dataset at a pre-specified kVp energy. The technique realizes monochromatic CT imaging and overcomes the beam hardening problem. PGPub US2020196973A1 [23] discloses an apparatus and method for a dual-energy CT system which uses sparse kVp-switching to collect data and a DL method to reconstruct the image. The deep network is fed with images that include artifacts and is trained to output images with little or no artifacts. Another deep network model can be trained to perform material decomposition from the artifact-free images.

Artifact reduction
Metal artifact reduction is one of the remaining problems in the CT field. European PGPub EP3743889A1 [24] discloses using DL to reduce metal artifacts. The neural network model is trained to generate a metal artifact image and generate a new image by subtracting the estimated metal artifacts from the input image. A relatively high number of patent documents target the same problem. Figure 7 lists some of them, ordered according to the number of forward citations.
The German PGPub DE102017219307B4 and Chinese PGPub CN109727203B [25,26] describe a system and method for compensating motion artifacts via ML. The technology relates to an automatic compensation method for motion artifacts in a medical image. The technology further relates to a method of automatically identifying motion artifacts, and includes a compensation unit, a learning device, and a controlling device for medical imaging. A number of patent documents address this problem, as shown in Fig. 8, ranked according to the number of forward citations.
The Japanese PGPub (JP2020534929A) [27] presents a DL-based scattering correction for X-ray imaging. A neural network is trained with a Mone Carlo simulated imaging data. This simulation includes at least the X-ray photon scattering mechanism that contaminates CT data with scattering noise. The neural network model learns  to remove the scattering noise in the CT data. After CT data are corrected by removing scatters in the projection space, the image reconstruction is performed to generate a scattering-corrected CT image. Scatter correction is also important for positron emission tomography (PET) imaging. A number of patent documents exist related to DL-based scatter correction for either CT or PET, as shown in Fig. 9.

Image analysis, radiomics and rawdiomics
International PGPub WO2018232388A1 [28] uses neural networks to integrate tomographic image reconstruction and radiomic analysis. CT screening, diagnosis or image analysis tasks are often performed with separate neural networks and algorithms. Integrating these elements into an end-to-end workflow may streamline the whole process and optimize the task-specific performance. As we know, while deep reconstruction is for image formation from raw data, image analysis or radiomics is for image analysis. Thus, the claimed integration of image reconstruction and image analysis is referred to as 'rawdiomics' , where 'rawd' means raw data, and 'i' indicates images or informatics. International PGPub WO2018/220089A1 [29] applies DL/ML to raw medical imaging data analysis for clinical decision support. The techniques are intended  for medical diagnosis from raw imaging data generated by a medical imaging machine with or without conventional image reconstruction.

Discussion and conclusions
First of all, we underline that although PatSeer ProX has provided useful information, we are still on the learning curve. We are not familiar with all of the functions and terminology of the PatSeer ProX tool. As a result, we believe that hidden information in the PatSeer dataset can be further mined. While we believe that we have obtained valuable data in this study, omissions and biases are unavoidable, due to the imperfect coverage of the database, dynamic nature of the field, and our limited capabilities. We apologize if we have made any misinterpretations.
Despite any problems possibly existing in this preliminary patent landscape analysis, we have learned significantly, facilitated by PatSeer ProX. Clearly, research and development in deep tomographic imaging has a strong momentum, engages both imaging companies and academic groups, and promises lasting impact on the further research and development as well as market and healthcare. Since the field of deep tomographic imaging is relatively young, more results and data are yet to be collected and analyzed to reveal the patent dynamics in terms of licensing, revenue, and translated outcomes. Also, it may be very informative to use PatSeer ProX, or similar tools, to identify and track emerging areas of AI-based imaging activities.
In conclusion, we have performed a preliminary landscape analysis on patent literature dedicated to deep tomographic imaging. Using the PatSeer ProX tool, we have systematically collected and analyzed relevant bibliometric data, and commented on representative deep tomographic imaging patent documents. Finally, we have discussed several issues and future work on these interesting topics.